Measuring k-Wise Independence of Streaming Data under L2 Norm

Measuring independence and k-wise independence is a fundamental problem that has multiple applications and it has been the subject of intensive research during the last decade (see, among others, the recent work of Batu, Fortnow, Fischer, Kumar, Rubinfeld and White [11] and of Alon, Andoni, Kaufman, Matulef, Rubinfeld and Xie [2] ). In the streaming environment, this problem was first addressed by Indyk and McGregor [44]. In this model the joint distribution is given empirically by a stream of elements, and the goal is to measure the distance between joint and product distribution under L distance. Indyk and McGregor give elegant solutions for estimating pairwise independence under L1 and L2 norms. The question of estimating k-wise independence on a stream of tuples, instead of pairs, is of central importance in multiple applications, where data typically comes with multiple attributes, such as database entries, minute-to minute changes in stock prices in a financial portfolio, and so on. Indyk and McGregor state, as an explicit open question in their paper, the problem of whether one can estimate k-wise independence on k-tuples for any k > 2. In this paper we answer the aforementioned open question of Indyk and McGregor [44] affirmatively for the L2 norm for any constant k. Our solution gives an (ǫ, δ)-approximation with O( 1 ǫ log 1 δ (logn+ logm)) memory bits, uses a single pass over the data and tolerates deletions. The main technical contribution of our paper is a novel combinatorial approach to analyzing second moment (i.e., variance) of dependent sketches. We believe that our method will be of independent interest. In our recent paper [15] we address the problem of measuring pairwise and k-wise independence under L1 norm. In [15] we use different methods which are not applicable to L2 norm.